SLOT : A research platform for investigating multimodal communication JAN

When two humans communicate with each other in a face-to-face situation, they do not only exchange speech. Other so-called channels or modalities, such as gaze (e.g., eye contact), facial expression, intonation, voice quality, and gestures (e.g., pointing) also play an important role in both the semantic and the socio-emotional aspects of communication (see, e.g., Ellsworth & Ludwig, 1972). Some authors have even claimed that nonverbal communication contributes more to the interpretation of communicative acts than does verbal information (Archer & Akert, 1977; Mehrabian & Wiener, 1967). Archer and Akert (1977) used recorded stimuli in which both the nonverbal information (voice quality) and the verbal information (semantic content) were systematically varied in emotional quality (hostile, neutral, or friendly). In judging the emotional quality of the stimuli in which the nonverbal and verbal qualities did not match, participants indeed attributed more importance to the nonverbal signal. Brown (1986) argued that when judges of emotional quality are presented with incongruous information, the nonverbal aspects of the communication are usually assumed to be under less conscious control by the speaker than the verbal content is, and, therefore, the nonverbal information is perceived to be a more reliable source than the verbal content (p. 498). Some authors have put forth the weaker claim that nonverbal communication is more suited for expressing feelings and affective information, whereas the content of speech is more often employed to refer to speaker-external events (Argyle, Salter, Nicolson, Williams, & Burgess, 1970). Later, however, Krauss, Apple, Morency, Wenzel, and Winton (1981), using naturalistic stimulus material, demonstrated that in judgments of the emotional quality of an exchange, judgments of the verbal content (presented in the form of transcripts) were a much more reliable source of information than those of video-only or content-filtered speech (presenting voice quality only) when each was compared with judgments of the full video and audio data, which served as a baseline. Independent of the contribution of nonverbal communication to the overall interpretation of communicative acts, there is no doubt that for the transmission of certain types of abstract information (e.g., conditionals, counterThis work was supported by Grant IST-2001-32311 from the European Union Project Conversational Multimodal Interaction with Computers (COMIC). The authors thank Jolanda Bource, Paula Dings, and Hanneke Ribberink for their meticulous transcription work. Correspondence concerning this article should be addressed to J. P. de Ruiter, Max Planck Institute for Psycholinguistics, P. O. Box 310, NL-6500 AH, Nijmegen, The Netherlands (e-mail: [email protected]). SLOT: A research platform for investigating multimodal communication